Skip to content

Provide optimized writers for OpenTelemetry's "trace.proto" wire protocol#11120

Open
mcculls wants to merge 18 commits intomasterfrom
mcculls/otlp-traces-proto
Open

Provide optimized writers for OpenTelemetry's "trace.proto" wire protocol#11120
mcculls wants to merge 18 commits intomasterfrom
mcculls/otlp-traces-proto

Conversation

@mcculls
Copy link
Copy Markdown
Contributor

@mcculls mcculls commented Apr 15, 2026 •
edited
Loading

What Does This Do

Uses a single temporary buffer as in #10983 to prepare message chunks at different nesting levels (resource / scope / span)

First we chunk all nested messages, i.e. span-links, for a given span. Once the span is complete we add the first part of the span message and its chunked links to the scoped chunks. Once the scope is complete we add the first part of the scoped spans message and all its chunks (span messages and their links) to the payload. Once all the span data has been chunked we add the enclosing resource metrics message to the start of the payload.

Multiple traces can be added to the collector before collecting them into a payload. Note that this payload is only valid for the calling thread until the next collection. Adding traces after collection automatically starts a new payload.

Motivation

Avoids need to use full protobuf library while keeping intermediate array creation to a minimum.

Additional Notes

OtlpTraceProtoTest was created with the help of Claude.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@mcculls mcculls added tag: do not merge Do not merge changes type: feature request inst: opentelemetry OpenTelemetry instrumentation labels Apr 15, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 15, 2026 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/otlp-traces-proto
git_commit_date 1776423906 1776452042
git_commit_sha d5d2097 97b5fc9
release_version 1.62.0-SNAPSHOT~d5d2097cb9 1.62.0-SNAPSHOT~97b5fc9e3d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776453914 1776453914
ci_job_id 1607130484 1607130484
ci_pipeline_id 108311916 108311916
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-vg2jvdsm 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-vg2jvdsm 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 63 metrics, 8 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064621
Total [baseline] (11.045 s) : 0, 11045400
Agent [candidate] (1.057 s) : 0, 1057313
Total [candidate] (11.093 s) : 0, 11093352
section appsec
Agent [baseline] (1.256 s) : 0, 1256345
Total [baseline] (11.119 s) : 0, 11119130
Agent [candidate] (1.254 s) : 0, 1253832
Total [candidate] (11.042 s) : 0, 11042396
section iast
Agent [baseline] (1.224 s) : 0, 1224079
Total [baseline] (11.282 s) : 0, 11282237
Agent [candidate] (1.224 s) : 0, 1223578
Total [candidate] (11.288 s) : 0, 11287970
section profiling
Agent [baseline] (1.184 s) : 0, 1183662
Total [baseline] (11.009 s) : 0, 11009444
Agent [candidate] (1.192 s) : 0, 1192130
Total [candidate] (11.019 s) : 0, 11019491
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.065 s -
Agent appsec 1.256 s 191.724 ms (18.0%)
Agent iast 1.224 s 159.457 ms (15.0%)
Agent profiling 1.184 s 119.04 ms (11.2%)
Total tracing 11.045 s -
Total appsec 11.119 s 73.729 ms (0.7%)
Total iast 11.282 s 236.836 ms (2.1%)
Total profiling 11.009 s -35.956 ms (-0.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent appsec 1.254 s 196.519 ms (18.6%)
Agent iast 1.224 s 166.265 ms (15.7%)
Agent profiling 1.192 s 134.817 ms (12.8%)
Total tracing 11.093 s -
Total appsec 11.042 s -50.956 ms (-0.5%)
Total iast 11.288 s 194.619 ms (1.8%)
Total profiling 11.019 s -73.861 ms (-0.7%) 
gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.242 ms) : 0, 1242
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (636.9 ms) : 0, 636900
BytebuddyAgent [candidate] (632.802 ms) : 0, 632802
AgentMeter [baseline] (29.795 ms) : 0, 29795
AgentMeter [candidate] (29.555 ms) : 0, 29555
GlobalTracer [baseline] (249.272 ms) : 0, 249272
GlobalTracer [candidate] (249.149 ms) : 0, 249149
AppSec [baseline] (32.527 ms) : 0, 32527
AppSec [candidate] (32.369 ms) : 0, 32369
Debugger [baseline] (59.969 ms) : 0, 59969
Debugger [candidate] (59.967 ms) : 0, 59967
Remote Config [baseline] (592.323 µs) : 0, 592
Remote Config [candidate] (596.144 µs) : 0, 596
Telemetry [baseline] (8.079 ms) : 0, 8079
Telemetry [candidate] (7.999 ms) : 0, 7999
Flare Poller [baseline] (10.008 ms) : 0, 10008
Flare Poller [candidate] (7.534 ms) : 0, 7534
section appsec
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (666.273 ms) : 0, 666273
BytebuddyAgent [candidate] (665.577 ms) : 0, 665577
AgentMeter [baseline] (12.359 ms) : 0, 12359
AgentMeter [candidate] (12.307 ms) : 0, 12307
GlobalTracer [baseline] (250.179 ms) : 0, 250179
GlobalTracer [candidate] (250.0 ms) : 0, 250000
IAST [baseline] (24.619 ms) : 0, 24619
IAST [candidate] (24.58 ms) : 0, 24580
AppSec [baseline] (186.263 ms) : 0, 186263
AppSec [candidate] (185.022 ms) : 0, 185022
Debugger [baseline] (66.401 ms) : 0, 66401
Debugger [candidate] (66.131 ms) : 0, 66131
Remote Config [baseline] (612.672 µs) : 0, 613
Remote Config [candidate] (604.0 µs) : 0, 604
Telemetry [baseline] (8.374 ms) : 0, 8374
Telemetry [candidate] (8.387 ms) : 0, 8387
Flare Poller [baseline] (3.5 ms) : 0, 3500
Flare Poller [candidate] (3.568 ms) : 0, 3568
section iast
crashtracking [baseline] (1.215 ms) : 0, 1215
crashtracking [candidate] (1.219 ms) : 0, 1219
BytebuddyAgent [baseline] (798.599 ms) : 0, 798599
BytebuddyAgent [candidate] (799.605 ms) : 0, 799605
AgentMeter [baseline] (11.534 ms) : 0, 11534
AgentMeter [candidate] (11.539 ms) : 0, 11539
GlobalTracer [baseline] (239.904 ms) : 0, 239904
GlobalTracer [candidate] (239.167 ms) : 0, 239167
IAST [baseline] (25.991 ms) : 0, 25991
IAST [candidate] (25.752 ms) : 0, 25752
AppSec [baseline] (31.654 ms) : 0, 31654
AppSec [candidate] (31.909 ms) : 0, 31909
Debugger [baseline] (65.604 ms) : 0, 65604
Debugger [candidate] (64.55 ms) : 0, 64550
Remote Config [baseline] (536.577 µs) : 0, 537
Remote Config [candidate] (533.48 µs) : 0, 533
Telemetry [baseline] (9.341 ms) : 0, 9341
Telemetry [candidate] (9.645 ms) : 0, 9645
Flare Poller [baseline] (3.637 ms) : 0, 3637
Flare Poller [candidate] (3.557 ms) : 0, 3557
section profiling
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (690.52 ms) : 0, 690520
BytebuddyAgent [candidate] (696.296 ms) : 0, 696296
AgentMeter [baseline] (9.171 ms) : 0, 9171
AgentMeter [candidate] (9.254 ms) : 0, 9254
GlobalTracer [baseline] (207.146 ms) : 0, 207146
GlobalTracer [candidate] (208.379 ms) : 0, 208379
AppSec [baseline] (32.816 ms) : 0, 32816
AppSec [candidate] (33.029 ms) : 0, 33029
Debugger [baseline] (65.794 ms) : 0, 65794
Debugger [candidate] (66.106 ms) : 0, 66106
Remote Config [baseline] (580.727 µs) : 0, 581
Remote Config [candidate] (572.888 µs) : 0, 573
Telemetry [baseline] (7.668 ms) : 0, 7668
Telemetry [candidate] (7.774 ms) : 0, 7774
Flare Poller [baseline] (3.508 ms) : 0, 3508
Flare Poller [candidate] (3.553 ms) : 0, 3553
ProfilingAgent [baseline] (94.138 ms) : 0, 94138
ProfilingAgent [candidate] (94.614 ms) : 0, 94614
Profiling [baseline] (94.705 ms) : 0, 94705
Profiling [candidate] (95.184 ms) : 0, 95184
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1050480
Total [baseline] (8.81 s) : 0, 8809977
Agent [candidate] (1.056 s) : 0, 1056271
Total [candidate] (8.826 s) : 0, 8825607
section iast
Agent [baseline] (1.233 s) : 0, 1232646
Total [baseline] (9.602 s) : 0, 9601623
Agent [candidate] (1.22 s) : 0, 1220004
Total [candidate] (9.565 s) : 0, 9564927
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.05 s -
Agent iast 1.233 s 182.166 ms (17.3%)
Total tracing 8.81 s -
Total iast 9.602 s 791.645 ms (9.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.22 s 163.733 ms (15.5%)
Total tracing 8.826 s -
Total iast 9.565 s 739.32 ms (8.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.217 ms) : 0, 1217
BytebuddyAgent [baseline] (630.715 ms) : 0, 630715
BytebuddyAgent [candidate] (631.879 ms) : 0, 631879
AgentMeter [baseline] (29.486 ms) : 0, 29486
AgentMeter [candidate] (29.544 ms) : 0, 29544
GlobalTracer [baseline] (247.393 ms) : 0, 247393
GlobalTracer [candidate] (248.663 ms) : 0, 248663
AppSec [baseline] (32.296 ms) : 0, 32296
AppSec [candidate] (32.477 ms) : 0, 32477
Debugger [baseline] (58.805 ms) : 0, 58805
Debugger [candidate] (58.937 ms) : 0, 58937
Remote Config [baseline] (596.858 µs) : 0, 597
Remote Config [candidate] (589.703 µs) : 0, 590
Telemetry [baseline] (8.02 ms) : 0, 8020
Telemetry [candidate] (8.766 ms) : 0, 8766
Flare Poller [baseline] (5.886 ms) : 0, 5886
Flare Poller [candidate] (8.117 ms) : 0, 8117
section iast
crashtracking [baseline] (1.247 ms) : 0, 1247
crashtracking [candidate] (1.243 ms) : 0, 1243
BytebuddyAgent [baseline] (809.588 ms) : 0, 809588
BytebuddyAgent [candidate] (797.867 ms) : 0, 797867
AgentMeter [baseline] (11.57 ms) : 0, 11570
AgentMeter [candidate] (11.564 ms) : 0, 11564
GlobalTracer [baseline] (238.759 ms) : 0, 238759
GlobalTracer [candidate] (238.288 ms) : 0, 238288
AppSec [baseline] (31.285 ms) : 0, 31285
AppSec [candidate] (32.736 ms) : 0, 32736
Debugger [baseline] (64.475 ms) : 0, 64475
Debugger [candidate] (62.95 ms) : 0, 62950
Remote Config [baseline] (540.116 µs) : 0, 540
Remote Config [candidate] (539.597 µs) : 0, 540
Telemetry [baseline] (9.343 ms) : 0, 9343
Telemetry [candidate] (9.41 ms) : 0, 9410
Flare Poller [baseline] (3.637 ms) : 0, 3637
Flare Poller [candidate] (3.597 ms) : 0, 3597
IAST [baseline] (25.892 ms) : 0, 25892
IAST [candidate] (25.712 ms) : 0, 25712
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/otlp-traces-proto
git_commit_date 1776423906 1776452042
git_commit_sha d5d2097 97b5fc9
release_version 1.62.0-SNAPSHOT~d5d2097cb9 1.62.0-SNAPSHOT~97b5fc9e3d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776454375 1776454375
ci_job_id 1607130487 1607130487
ci_pipeline_id 108311916 108311916
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-rzo5wxhs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-rzo5wxhs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 6 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast_FULL:high_load better
[-542.828µs; -288.332µs] or [-9.682%; -5.143%]		 better
[-1234.896µs; -644.628µs] or [-9.303%; -4.856%]		 unstable
[-31.068op/s; +137.756op/s] or [-4.200%; +18.621%]		 5.191ms 12.334ms 793.125op/s 5.607ms 13.274ms 739.781op/s
scenario:load:petclinic:appsec:high_load better
[-2.013ms; -1.262ms] or [-10.231%; -6.415%]		 better
[-2.880ms; -1.458ms] or [-9.081%; -4.598%]		 unstable
[-8.310op/s; +44.185op/s] or [-3.541%; +18.830%]		 18.035ms 29.541ms 252.594op/s 19.672ms 31.710ms 234.656op/s
scenario:load:petclinic:iast:high_load better
[-2.113ms; -1.629ms] or [-10.949%; -8.444%]		 better
[-2.932ms; -1.652ms] or [-9.496%; -5.351%]		 unstable
[-2.862op/s; +50.924op/s] or [-1.202%; +21.383%]		 17.424ms 28.585ms 262.188op/s 19.295ms 30.877ms 238.156op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.261 ms) : 1248, 1273
.   : milestone, 1261,
iast (3.386 ms) : 3338, 3434
.   : milestone, 3386,
iast_FULL (6.254 ms) : 6190, 6318
.   : milestone, 6254,
iast_GLOBAL (3.61 ms) : 3551, 3670
.   : milestone, 3610,
profiling (2.212 ms) : 2192, 2232
.   : milestone, 2212,
tracing (1.955 ms) : 1938, 1972
.   : milestone, 1955,
section candidate
no_agent (1.244 ms) : 1231, 1258
.   : milestone, 1244,
iast (3.41 ms) : 3358, 3461
.   : milestone, 3410,
iast_FULL (5.829 ms) : 5771, 5888
.   : milestone, 5829,
iast_GLOBAL (3.645 ms) : 3590, 3701
.   : milestone, 3645,
profiling (2.241 ms) : 2219, 2263
.   : milestone, 2241,
tracing (1.925 ms) : 1909, 1941
.   : milestone, 1925,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.261 ms [1.248 ms, 1.273 ms] -
iast 3.386 ms [3.338 ms, 3.434 ms] 2.125 ms (168.5%)
iast_FULL 6.254 ms [6.19 ms, 6.318 ms] 4.993 ms (396.0%)
iast_GLOBAL 3.61 ms [3.551 ms, 3.67 ms] 2.349 ms (186.3%)
profiling 2.212 ms [2.192 ms, 2.232 ms] 950.835 µs (75.4%)
tracing 1.955 ms [1.938 ms, 1.972 ms] 694.233 µs (55.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.244 ms [1.231 ms, 1.258 ms] -
iast 3.41 ms [3.358 ms, 3.461 ms] 2.165 ms (174.0%)
iast_FULL 5.829 ms [5.771 ms, 5.888 ms] 4.585 ms (368.5%)
iast_GLOBAL 3.645 ms [3.59 ms, 3.701 ms] 2.401 ms (192.9%)
profiling 2.241 ms [2.219 ms, 2.263 ms] 996.765 µs (80.1%)
tracing 1.925 ms [1.909 ms, 1.941 ms] 680.475 µs (54.7%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.95 ms) : 17770, 18129
.   : milestone, 17950,
appsec (19.891 ms) : 19684, 20099
.   : milestone, 19891,
code_origins (18.151 ms) : 17971, 18332
.   : milestone, 18151,
iast (19.596 ms) : 19400, 19793
.   : milestone, 19596,
profiling (19.314 ms) : 19117, 19510
.   : milestone, 19314,
tracing (17.654 ms) : 17479, 17829
.   : milestone, 17654,
section candidate
no_agent (17.38 ms) : 17201, 17559
.   : milestone, 17380,
appsec (18.475 ms) : 18285, 18664
.   : milestone, 18475,
code_origins (17.994 ms) : 17816, 18172
.   : milestone, 17994,
iast (17.798 ms) : 17622, 17973
.   : milestone, 17798,
profiling (18.969 ms) : 18776, 19162
.   : milestone, 18969,
tracing (17.901 ms) : 17725, 18077
.   : milestone, 17901,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.95 ms [17.77 ms, 18.129 ms] -
appsec 19.891 ms [19.684 ms, 20.099 ms] 1.941 ms (10.8%)
code_origins 18.151 ms [17.971 ms, 18.332 ms] 201.689 µs (1.1%)
iast 19.596 ms [19.4 ms, 19.793 ms] 1.647 ms (9.2%)
profiling 19.314 ms [19.117 ms, 19.51 ms] 1.364 ms (7.6%)
tracing 17.654 ms [17.479 ms, 17.829 ms] -295.832 µs (-1.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.38 ms [17.201 ms, 17.559 ms] -
appsec 18.475 ms [18.285 ms, 18.664 ms] 1.095 ms (6.3%)
code_origins 17.994 ms [17.816 ms, 18.172 ms] 613.897 µs (3.5%)
iast 17.798 ms [17.622 ms, 17.973 ms] 417.914 µs (2.4%)
profiling 18.969 ms [18.776 ms, 19.162 ms] 1.589 ms (9.1%)
tracing 17.901 ms [17.725 ms, 18.077 ms] 520.592 µs (3.0%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/otlp-traces-proto
git_commit_date 1776423906 1776452042
git_commit_sha d5d2097 97b5fc9
release_version 1.62.0-SNAPSHOT~d5d2097cb9 1.62.0-SNAPSHOT~97b5fc9e3d
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1776454311 1776454311
ci_job_id 1607130489 1607130489
ci_pipeline_id 108311916 108311916
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-aj015hbg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-aj015hbg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.49 ms) : 1478, 1501
.   : milestone, 1490,
appsec (3.835 ms) : 3611, 4060
.   : milestone, 3835,
iast (2.285 ms) : 2215, 2355
.   : milestone, 2285,
iast_GLOBAL (2.335 ms) : 2264, 2405
.   : milestone, 2335,
profiling (2.112 ms) : 2056, 2167
.   : milestone, 2112,
tracing (2.081 ms) : 2027, 2135
.   : milestone, 2081,
section candidate
no_agent (1.492 ms) : 1480, 1503
.   : milestone, 1492,
appsec (3.82 ms) : 3598, 4043
.   : milestone, 3820,
iast (2.278 ms) : 2208, 2347
.   : milestone, 2278,
iast_GLOBAL (2.324 ms) : 2254, 2394
.   : milestone, 2324,
profiling (2.11 ms) : 2055, 2166
.   : milestone, 2110,
tracing (2.085 ms) : 2031, 2139
.   : milestone, 2085,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.49 ms [1.478 ms, 1.501 ms] -
appsec 3.835 ms [3.611 ms, 4.06 ms] 2.346 ms (157.5%)
iast 2.285 ms [2.215 ms, 2.355 ms] 795.365 µs (53.4%)
iast_GLOBAL 2.335 ms [2.264 ms, 2.405 ms] 845.3 µs (56.7%)
profiling 2.112 ms [2.056 ms, 2.167 ms] 622.035 µs (41.8%)
tracing 2.081 ms [2.027 ms, 2.135 ms] 591.424 µs (39.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.492 ms [1.48 ms, 1.503 ms] -
appsec 3.82 ms [3.598 ms, 4.043 ms] 2.329 ms (156.1%)
iast 2.278 ms [2.208 ms, 2.347 ms] 786.369 µs (52.7%)
iast_GLOBAL 2.324 ms [2.254 ms, 2.394 ms] 832.338 µs (55.8%)
profiling 2.11 ms [2.055 ms, 2.166 ms] 618.898 µs (41.5%)
tracing 2.085 ms [2.031 ms, 2.139 ms] 593.452 µs (39.8%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~97b5fc9e3d, baseline=1.62.0-SNAPSHOT~d5d2097cb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.518 s) : 15518000, 15518000
.   : milestone, 15518000,
appsec (14.806 s) : 14806000, 14806000
.   : milestone, 14806000,
iast (18.361 s) : 18361000, 18361000
.   : milestone, 18361000,
iast_GLOBAL (17.898 s) : 17898000, 17898000
.   : milestone, 17898000,
profiling (14.997 s) : 14997000, 14997000
.   : milestone, 14997000,
tracing (14.811 s) : 14811000, 14811000
.   : milestone, 14811000,
section candidate
no_agent (14.996 s) : 14996000, 14996000
.   : milestone, 14996000,
appsec (15.012 s) : 15012000, 15012000
.   : milestone, 15012000,
iast (18.57 s) : 18570000, 18570000
.   : milestone, 18570000,
iast_GLOBAL (18.251 s) : 18251000, 18251000
.   : milestone, 18251000,
profiling (14.956 s) : 14956000, 14956000
.   : milestone, 14956000,
tracing (15.191 s) : 15191000, 15191000
.   : milestone, 15191000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.518 s [15.518 s, 15.518 s] -
appsec 14.806 s [14.806 s, 14.806 s] -712.0 ms (-4.6%)
iast 18.361 s [18.361 s, 18.361 s] 2.843 s (18.3%)
iast_GLOBAL 17.898 s [17.898 s, 17.898 s] 2.38 s (15.3%)
profiling 14.997 s [14.997 s, 14.997 s] -521.0 ms (-3.4%)
tracing 14.811 s [14.811 s, 14.811 s] -707.0 ms (-4.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.996 s [14.996 s, 14.996 s] -
appsec 15.012 s [15.012 s, 15.012 s] 16.0 ms (0.1%)
iast 18.57 s [18.57 s, 18.57 s] 3.574 s (23.8%)
iast_GLOBAL 18.251 s [18.251 s, 18.251 s] 3.255 s (21.7%)
profiling 14.956 s [14.956 s, 14.956 s] -40.0 ms (-0.3%)
tracing 15.191 s [15.191 s, 15.191 s] 195.0 ms (1.3%)

@mcculls mcculls force-pushed the mcculls/otlp-traces-proto branch 2 times, most recently from 583dc0c to 4adb56e Compare April 15, 2026 17:53
@mcculls mcculls changed the title WIP: Provide optimized writers for OpenTelemetry's "trace.proto" wire protocol Provide optimized writers for OpenTelemetry's "trace.proto" wire protocol Apr 15, 2026
@mcculls mcculls marked this pull request as ready for review April 15, 2026 17:56
@mcculls mcculls requested a review from a team as a code owner April 15, 2026 17:56
@mcculls mcculls requested review from mtoffl01 and ygree and removed request for ygree April 15, 2026 17:56
@mcculls mcculls removed the tag: do not merge Do not merge changes label Apr 15, 2026
mtoffl01
mtoffl01 reviewed Apr 16, 2026
View reviewed changes
Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/trace/OtlpTraceProtoCollector.java Outdated
Comment on lines +59 to +65
/**
* Collects trace spans and marshalls them into a chunked payload.
*
* <p>This payload is only valid for the calling thread until the next collection.
*/
@Override
public OtlpPayload collectSpans(List<DDSpan> spans) {
Copy link
Copy Markdown
Contributor

@mtoffl01 mtoffl01 Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is List<DDSpan> spans expected to be spans from a single trace? If so, each collectSpans call produces a full TracesData envelope with resource and scope wrappers per trace. This doesn't seem optimal and differs from the Datadog/msgpack implementation? Unless the expectation is that the eventual OtlpWriter will accumulated completed traces and call this once per flush cycle with a combined span list (although that can't be right based on the MetaWriter, which expects just a single trace at a time).

Copy link
Copy Markdown
Contributor Author

@mcculls mcculls Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good point - on reflection I'll change this to add a flush method so we can accumulate trace chunks over multiple calls.

Copy link
Copy Markdown
Contributor Author

@mcculls mcculls Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I've updated the collector API so it has two methods:

  • addTrace(spans) which adds a trace to the collector
  • collectTraces() which marshals the collected spans into a payload

This should allow its use as a replacement PayloadDispatcher, which means we can re-use more of the existing remote writer code.

Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/trace/OtlpTraceProto.java Outdated
mcculls added 10 commits April 17, 2026 12:22
@mcculls mcculls force-pushed the mcculls/otlp-traces-proto branch from ab2ef0b to 7cdfed7 Compare April 17, 2026 11:53
@mcculls mcculls requested a review from a team as a code owner April 17, 2026 11:53
@mcculls mcculls requested review from dougqh and removed request for a team April 17, 2026 11:53
dougqh
dougqh reviewed Apr 17, 2026
View reviewed changes
Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/trace/OtlpTraceProto.java Outdated
dougqh
dougqh reviewed Apr 17, 2026
View reviewed changes
Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpCommonProto.java Outdated
dougqh
dougqh reviewed Apr 17, 2026
View reviewed changes
Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/trace/OtlpTraceProto.java Outdated
dougqh
dougqh reviewed Apr 17, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@dougqh dougqh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude caught a couple issues...

  • NPE and ClassCastException

Since I'm off next week, I'm not going to "request changes".
I'll just trust those get fixed and let someone else do the final review.

Also, added one key performance suggestion around use of forEach.

And here are couple more Claude reported that I'll leave to your discretion...

  1. Config.get().getServiceName() on every span — OtlpTraceProto.java:137
    if (!Config.get().getServiceName().equalsIgnoreCase(span.getServiceName())) {
    Cache the default service name (ideally as a UTF8BytesString for cheap equality). This runs for every span in every payload.

  2. recordMessage allocates a fresh ByteBuffer + backing array per chunk — OtlpCommonProto.java:126-140
    Every span, every link, every scope prefix gets its own heap allocation. Precisely-sized allocations are nice but total allocation count scales with the
    chunk count. If profiling shows GC pressure, a small reusable scratch arena that hands out slices (or an OtlpPayload that owns a large backing buffer with
    offset/length pairs) would eliminate most of this. Trade-off is lifetime complexity, so only worth it if measurements show it matters.

@mcculls
Copy link
Copy Markdown
Contributor Author

mcculls commented Apr 17, 2026

recordMessage allocates a fresh ByteBuffer + backing array per chunk — OtlpCommonProto.java:126-140
Every span, every link, every scope prefix gets its own heap allocation. Precisely-sized allocations are nice but total allocation count scales with the
chunk count. If profiling shows GC pressure, a small reusable scratch arena that hands out slices (or an OtlpPayload that owns a large backing buffer with
offset/length pairs) would eliminate most of this. Trade-off is lifetime complexity, so only worth it if measurements show it matters.

Yes, sadly this is the nature of heavily nested protobuf messages (the protobuf manual says to avoid too much nesting)

It means that before we can write out a span we need to know its exact message size. And because the size field is written out with varint encoding, different sizes take up different amounts of space even for just the size field. This means we can't leave placeholders to come back and write the size, because we don't know how many bytes the size field will take up, and if we estimate wrong then we'd then have to shift different chunks around multiple times. Ditto for any links it has, because the OTel proto spec says they're nested inside spans - and span-links may have attributes, which means another level of nested message!

You could process traces twice - once to size everything, and again to write it out - but the book-keeping needed for that gets complicated, and you're doubling the CPU time doing two passes.

Initial benchmarking showed we're allocating less than OTel with the current approach, mainly because we re-use the same buffer for doing the initial writes before recording each message slice. But I might look into pooling of slices to reduce churn.

@mcculls mcculls force-pushed the mcculls/otlp-traces-proto branch from 97b5fc9 to e77fe7e Compare April 17, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dougqh dougqh dougqh left review comments

@mtoffl01 mtoffl01 mtoffl01 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

inst: opentelemetry OpenTelemetry instrumentation type: feature request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mcculls @dougqh @mtoffl01